3-D Immersion Technology in a Virtual Store

ABSTRACT

A system creates an enhanced virtual shopping experience, and includes means for generating a 3-D mesh model of a person&#39;s body contours; means for digitizing the three-dimensional shape of a piece of clothing when worn by the correspondingly sized body; means for measuring surface friction and intensity/frequency of vibrations produced during sliding tactile contact with the material of the clothing, for digitally approximating a material&#39;s texture; and a means for obtaining three dimensional visual images of the interior of a store. A 3-D headset is configured to receive the three-dimensional visualization of the store interior, and provide it to a person that may also wear a wired glove and body suit configured to provide tactile feedback or feedback through the use of electrodes. A software algorithm digitally inserts the person&#39;s mesh model into the three-dimensional store visualization, and allows manipulation of it therein, including realistic simulated contact of objects therein using reverse-electro-vibrations sent to the body suit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/321,984, filed Jul. 2, 2014, which claims priority on U.S.Provisional Application Ser. No. 61/842,020, filed on Jul. 2, 2013, thedisclosures of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to improvements in online shopping withina virtual mall, and more particularly to a method and system which arecapable of providing a shopper with a more realistic shoppingexperience. The virtual environment of the present invention also hasadvantageous use with respect to touring or viewing in general, such asfor virtual visiting of museums, or for virtually experiencingconcerts/plays, or even for virtually experiencing space travel, etc.

BACKGROUND OF THE INVENTION

The origins of E-commerce can be traced back to early direct marketingto customers by phone (“telemarketing”), and to T.V. commercials thatadvertised products that could be purchased from home by calling a phonenumber on the television screen, which eventually developed intohalf-hour or hour-long infomercials. However, the birth of theworld-wide-web in the early 1990s provided even greater opportunitiesfor e-commerce, and fostered the creation of the online retail giantAmazon.com®, and also the multinational online consumer-to-consumerauction provider eBay®. As technology breakthroughs occur, theopportunities for its application to further improve the electronicshopping experience continually evolve.

Many bricks and mortar clothing retailers now have websites that permitviewing of their products in each of the available colors, andadditionally list the available sizes. There are also many inventionsthat have sought to further improve the online marketing of retailproducts by seeking to add particular enhancements to the virtualexperience. For example, U.S. Pat. No. 6,307,568 to Rom for “VirtualDressing over the Internet” teaches “a system for displaying garments .. . as though the garments were being draped over the body of a user,”by fitting “articles of clothing to an image of a user,” using “apicture of the user” and “critical points” that are “taken from theimage of the user, and are used to adjust the spatial configuration ofthe clothing.” In addition, U.S. Pat. No. 7,149,665 to Feld teaches that“Virtual two-dimensional patterns representing different portions of thewear article are assembled into a virtual three-dimensional weararticle,” and furthermore teaches that a “material type is associatedwith one or more of the virtual patterns and the virtualthree-dimensional wear article,” permitting “the virtual wear article tostretch, flex, sag, etc., on the virtual model to better approximate thereal-life fit and look of the wear article on an object during design ofthe wear article.”

The present invention seeks to dramatically improve the realism of thevirtual shopping experience beyond these and other inventions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an exemplary computing unit capable ofrunning the software of the current invention, and may include cellularphones, personal computers and other wired/wireless smart devices, andwhich may be capable of interacting with external peripherals and otherdevices over the internet.

DETAILED DESCRIPTION OF THE INVENTION

There are many consumers who enjoy traveling to a retail store,particularly at a large shopping mall, so that they may try on clothingto see if the size is correct and if the particular cut by designer fitstheir frame well, and also to see if they find the visual appearance ofthe garment aesthetically pleasing when viewed on their person.Moreover, many consumers also assess the quality and/or desirability ofthe clothing based upon the feel of the material on their body, and thematerial's texture when rubbed between the person's thumb andforefinger. In addition, some consumers like to hold the garment infront of them using their hands to assess its weight and to see howsheer it may or may not be, and also to see how it tends to generallydrape. They may also wish to examine the interior lining of the item,and to assess the overall quality of manufacture of the garment.

However, despite these advantages of shopping at a bricks and mortarretailer, for many consumers proximity and scheduling constraints oftenlimit when they may have a sufficient amount of time to make the commutethereto, and to enjoy the shopping experience without unnecessary haste.The problem is exacerbated for consumers who live in rural areas thatare at considerable distances away from particular retailers.

The present invention provides the ability for the consumer to enjoy theabove described advantages attained when actually shopping at a bricksand mortar clothing retailer, but while instead conveniently shoppingfrom home, using the enhanced virtual shopping experience disclosedherein.

There are several aspects to the present invention, which are joinedthrough the application of particular technology to provide for anenhanced shopping experience. A first aspect of the present inventionpertains to the particular user who will be doing the shopping. In orderfor the person to accurately assess the fit of a particularmanufacturer's clothing upon their body, their own physical form mustfirst be digitized. Three-dimensional (3-D) shape information may begathered to generate a simulated 3-D mesh model of their body contoursusing one of several methods. In a first embodiment for producing suchdigital definition, a 3-D model may be created using a contact 3-Dscanner probe that physically touches the object—the person, whileresting upon a precision flat surface plate. Another method, which wouldbe much less invasive for the average consumer, entails creatingthree-dimensional models using 2-D photographs and specialized software,such as, for example, the object modeler available from 3dsom.com.Alternatively, 3-D modeling may be obtained through the use of anon-contact remote scanning system, such as LiDAR (e.g., light-radar),where a narrow pulsed laser beam is used to illuminate the target (theperson's body), and the reflected light is analyzed, often with a gatedcamera, to produce the three-dimensional definition. The 3-D data may bepreferably utilize X3D, which is an ISO standard XML-based file formatfor 3D computer graphics that permits humanoid animation, and is thesuccessor to the Virtual Reality Modeling Language (VRML).

Although photographs may also suffice, the latter method would be wellsuited for quick 3-D imaging of the person who may desirably be posed atmultiple standard positions (e.g., standing with arms at side and legstogether, standing with arms bent, standing with arms outstretched,standing with arms crossed; standing with legs apart as if walking;sitting with arms on lap, sitting with arms resting on bent . . . ). Theimagery may be updated to adjust the 3-D mesh model if the consumergains or loses weight, or otherwise grows, particularly if the consumeris a young child.

Another aspect of the present invention involves the modeling of thegarments of a particular clothing manufacturer. This may be obtainedusing a clothes measurement system by three-dimensional digitizing ofthe shape of each piece of clothing, as worn by the ideal sized body ormannequin, and with the input of design tolerances for the manufacturingof each garment type/size. Additional imaging may be produced tocharacterize the garment from all angles, including when looking at itsinterior, or when holding it up in front of a prospective purchaser. Thetexture of the material of each garment may be characterized using, forexample, an artificial finger. The artificial finger may be equippedwith strain gauges adapted to measure surface friction and to measurethe different intensities and frequencies of the vibrations producedduring certain tactile tasks performed by the finger upon the material,as disclosed, for example, in “Material Classification by TactileSensing Using Surface Texture,” by Jamali, N, and Sammut, C., IEEETransactions on Robotics, 508-521 (2011), the disclosures of which areincorporated herein by reference.

A further aspect of the present invention is that 3-D imagery may alsobe produced, using any of the above methods, for the interior of thebricks and mortar store for each retailer, which need not necessarily bea clothing retailer.

Finally, the software of the present invention may be configured toprovide this imagery to the user, who may wear 3-D goggles or a headset,in order to produce a three-dimensional visualization of the interior ofthe store, including all of the garments and other items for purchasethat are located therein. A series of cameras within participatingstores may be used to obtain visual images of the actual bricks andmortar retail environment, images which may be received and utilized bythe head set, so the user may engage virtually in a store shoppingexperience. The virtual shopping experience may either be a real-timevirtual experience (i.e., live store images are streamed to the headset), or a virtual experience that lags behind the real-time storeactivity (i.e., images recorded from an earlier time in the day areused). The user may thereby immerse themselves within the virtual storeand interact therein, in one of several ways. In the first method, theperson may interact using an Avatar, which is their three-dimensionalcomputer representation created by the mesh model, so the user would bea 3^(rd) person viewer.

The user may utilize any suitable digital pointing device for actuatingthe movements of the Avatar, such as a mouse, or a joystick, aspaceball, or a spacemouse. The customer Avatar may interact with asalesperson Avatar that may service the virtual shopping market the sameas they would provide in-person service within the bricks and mortarstore. When the virtual shopper sees an item of interest, a shirt forexample, the software of the present invention enables their Avatar toaccess the computer modeled shirt, in order to hold it up, to manipulateit and examine it as if they were physically in the bricks and mortarstore, and to try it on virtually, by placing it over thecomputer-generated Avatar of themselves, to obtain a realistic andlife-like perspective of how the real garment would fit on their actualbody. Animation may be used to enhance the movements of the Avatar, andthe robing and disrobing of the particular garment. This 3^(rd) personexperience may be elevated to a 1^(st) person experience through the useof mirrors and software that may trick the system and user. Similarly,the inputs to the system made by the user may also be at a higher lever,whereby instead of using a pointing device that is manipulated directlyby the user, the user's movements/responses within the virtualenvironment are obtained by cameras that monitor the user's actualmovements, and automatically input them into the virtual imagery. Thecameras can be an HD and/or a 3D camera. A connect camera or improvedversions thereof can also be used. The cameras, particularly for touristlocations and sporting events can include traffic and utility cameras

A higher level virtual experience may be obtained for the virtualshopper using additional technology. The person, in addition to wearingthe 3-D goggles, may also be wearing a wired glove and body suit. Thewired glove and body suit, which may be only for the upper torso(shirts), or only for the lower torso (slacks), or for both (e.g., ajumpsuit, or a shirt and slacks), may incorporate therein the latestgeneration of haptics—a tactile feedback technology. The haptics utilizereverse-electro-vibration, whereby a current is sent to a portion of thesuit to correspond to the object they are wearing or touching, to createthe appropriate sensation of friction, depending on the shape,frequency, and amplitude of the signal, as it corresponds to themeasured characteristics of the garment at that location. The hapticstechnology may thus convey to the person the nature of not only thevisual texture of the material of the shirt being “held” or “worn” inthe virtual shopping store, but also its tactile texture, by holding thevirtual material between their gloved fingers to virtually sense itssoftness, or coarseness, etc.

This technology also enables a user to interact in real time, and in apseudo 3-D virtual environment that, because of the goggles, the wiredbody suit, and the software, may appear to the user to be a first personexperience. That is, the user may look at the bodysuit that they arewearing, through the goggles, and the software may present theappropriate representation of the clothing (the shirt) thereon, and theperson in essence becomes the Avatar, and does not see a computerrepresentation of the clothing upon a pure computer-generated image(3^(rd) person viewing), but instead sees an 1 person representation ofthe garment upon their body, in a convergence of the virtual world andthe real world.

If the size of the garment is not satisfactory, other sizes of the sameshirt may similarly be worn in the mixed real/virtual world, as well asother shirts by other designers. When the consumer decides that thegarment fits appropriately and desires to acquire it, he/she mayinteract with the virtual store to purchase the item using a credit cardor any other means of e-commerce (Paypal®, etc.), and the retail storemay then process the transaction and mail the product to the consumer.

An even higher level of a virtual experience may also be obtained byutilizing 3D stereo glasses instead of the goggles, and the user may doso within a cave automatic virtual environment (CAVE) that may comprisea 6-sided cube. Multiple projectors may simultaneously projectcoordinated images to each of the six sides of the cube to form virtualsurroundings for the participant. The 3D glasses may permit theparticipant to view the 2D images projected upon the sides of the cubeas three-dimensional imagery. The participant's actual movements withinthe cube may be tracked by sensors, which may be coupled through amicroprocessor of a computing system to the projectors, and serves tomake corresponding adjustments to the visual image for each suchmovement to provide a proper view of the virtual reality. Softwarerunning on the computing system may be based upon scene renderingavailable using OpenScenegraph—an open source. 3D graphics API writtenin C++, or using OpenGL Performer, which is a free commercial library ofutility code that overlays the source code, OpenGL. Proprietary APIsoftware may also be developed to support the specific functionality ofthe invention described herein, which may also include the introductionof scents into the CAVE to add to the virtual experience. The scentscould be real, or instead, they may be created digitally, such as withthe digital scent device known as iSmell, which is being developed byDigiScents in Oakland Calif.

It should also be noted that the visual effects of the cube in the CAVEenvironment may comparably be achieved using the headset discussedhereinabove, whereby the headset provides imagery that encompasses theuser's entire field of view. The headset may also include stereoheadphones to provide the corresponding audio inputs to the user, andmay also incorporate the iScent device therein to complete the virtualexperience for the headset wearer. A headset microphone may be utilizedto permit two-way communication between the wearer and the personnelconducting the virtual experience.

The computing system may also control an audio system that providesrealistic sound inputs to the participant. The audio system may utilizea series of speakers on each side of the cube, where the intensity andtype of sounds emanating from each speaker may be controlled andmodified by the computing system to provide realistic sound effects,which may be coordinated with the projected imagery and theparticipant's movements. Instead of speakers, earphones mayalternatively be utilized, and the computing system would then need toadjust the sounds to each ear to mimic the sound differential to eachside that would be indicative of its origin/position.

It should be noted that the technology disclosed herein is suitable forvarious other applications, such as for virtual tourism. Imagery may beproduced for an entire city, or regions of a city, or of museumexhibits, which may be visited in a similar fashion, with the userseeing and experiencing the scene in real time. Other applications forthe invention include sporting events, concerts etc.

Software of the present invention may run on a computer, a server, atablet, a cell phone, or other smart device, so a description of such anaccessorized exemplary computer system is hereinafter disclosed, eventhough a particular embodiment may not require all of the describedcomponents. Exemplary computer system 200 is shown schematically in FIG.1, and which may comprise computing unit 201 interacting with externalperipherals 202, such as a separate touch screen display 244, andinteracting with network resources 203, including use of the internet261, and other computers, which may be first and second laptop computers262/263, a tablet, a smart phone etc.

The computing unit 201 may include a data bus 224 for communicatinginformation across and among various parts of computing unit 201, and acentral processing unit, which may be a microprocessor (hereinafter“processor” or “CPU”) 222 coupled with a bus 224 for processinginformation and performing other computational and control tasks.Computing unit 201 may also include a volatile storage 225, such as arandom access memory (RAM) or other dynamic storage device, coupled tobus 224 for storing various information as well as instructions to beexecuted by processor 222. The RAM may be Dynamic Random Access Memory(DRAM), or Static RAM (SRAM), or any other similar type of RAM known inthe art. The volatile storage 225 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions by processor 222. Computing unit 201 may further include aread only memory (ROM) or an erasable programmable memory (EPROM) 227 orother static storage device coupled to bus 224 for storing staticinformation and instructions for processor 222, such as basicinput-output system (BIOS), as well as various system configurationparameters. A persistent storage device or non-volatile memory 226, suchas a magnetic disk, optical disk, or solid-state flash memory device maybe provided and may be coupled to bus 224 for storing information andinstructions.

Computing unit 201 may be coupled via bus 224 to an integral display221, possibly a touch-screen display, for use in displaying informationto a user. If desired, computing unit 201 may be coupled via bus 224 toan external display screen 244. An external input device 243 (e.g., astandard keyboard) may be coupled to bus 224 for communicatinginformation and command selections to processor 222. A cursor controldevice 242, such as a mouse, a trackball, or cursor direction keys, maybe used for communicating direction information and command selectionsto processor 222 and for controlling cursor movement on display 244. Anexternal storage device 241 may be connected to the computing unit 201via bus 224 to provide an extra or removable storage capacity for thecomputing unit 201, which may be used to facilitate exchange of datawith other computer systems.

Some of the techniques herein may be performed by computing unit 201 inresponse to processor 222 executing one or more sequences of one or moreinstructions contained in the volatile memory 225. Execution of thesequences of instructions contained in a memory may cause processor 222to perform the process steps described herein. In alternativeembodiments, specific hard-wired digital circuitry may be used in placeof, or in combination with, software instructions to implement theinvention.

The term “computer-readable medium” as used herein refers to any mediumthat participates in providing instructions to processor 222 forexecution. The computer-readable medium is just one example of amachine-readable medium, which may carry instructions for implementingany of the methods and/or techniques described herein. Various forms ofcomputer readable media may be involved in carrying one or moresequences of one or more instructions to processor 222 for execution,including non-volatile media (storage device 226), and volatile media(storage device 225). Common forms of computer-readable media include,for example, a floppy disk, a hard disk, magnetic tape, or any othermagnetic medium, a CD-ROM, any other optical medium, a RAM, a PROM, anEPROM, a flash drive, and a memory card.

The computing unit 201 may thus also include a communication interface,such as network interface card 223 coupled to the data bus 222.Communication interface 223 may provide a two-way data communicationcoupling to a network link that may be connected to a local network. Forexample, communication interface 223 may be an integrated servicesdigital network (ISDN) card or a modem to provide a data communicationconnection to a corresponding type of telephone line, or it may be alocal area network interface card (LAN NIC) to provide a datacommunication connection to a compatible LAN.

Network link 223 also typically provides data communication to othernetwork resources. For example, the network link may provide aconnection over the internet 261 to the world-wide-web. Thus, thecomputing unit 201 can access resources located anywhere using theInternet 261. Also, the computing unit 201 may also be accessed by othercomputers (e.g. 262-263), generally with permission, and which may belocated anywhere with access to the internet 261.

The examples and descriptions provided merely illustrate a preferredembodiment of the present invention. Those skilled in the art and havingthe benefit of the present disclosure will appreciate that furtherembodiments may be implemented with various changes within the scope ofthe present invention. Other modifications, substitutions, omissions andchanges may be made in the design, size, materials used or proportions,operating conditions, assembly sequence, or arrangement or positioningof elements and members of the preferred embodiment without departingfrom the spirit of this invention.

We claim:
 1. A system to enable an enhanced virtual shopping experience,said system comprising: means for generating a 3-D mesh model of aperson's body contours; means for digitizing the three-dimensional shapeof a piece of clothing when worn by the correspondingly sized body;means for measuring the surface friction and the intensity and frequencyof the vibrations produced during sliding tactile contact with thematerial of the piece of clothing, for digitally approximating a textureof the material; means for obtaining three dimensional visual images ofthe interior of a store; a 3-D headset configured to receive thethree-dimensional visualization of the interior of the store, and toprovide the visualization to the wearer; a wired glove and body suitconfigured to provide feedback to the wearer; a software algorithmconfigured to enable said 3-D mesh model of the person's body to bedigitally inserted into said three-dimensional visualization of theinterior of the store; said software algorithm further configured tomanipulate said 3-D mesh model within the store, and to provide acurrent to a portion of said glove or suit to providereverse-electro-vibration thereto, to correspond to the object that isin contact with the wearer, to create the appropriate sensation ofcontact that simulates actual contact.
 2. The system according to claim1 wherein the shopping experience can be enhanced through theinteraction or use of social media.
 3. The system according to claim 1wherein an Avatar is generated by the system
 4. The system according toclaim 3 wherein one Avatar can interact and shop with other Avatars. 5.The system according to claim 4 wherein the Avatars can be manipulatedthrough the use of social media.
 6. The system according to claim 1wherein the model can be viewed in real time.
 7. The system according toclaim 1 wherein the model can be viewed in lag time.
 8. The systemaccording to claim 1 where the user is provided with a first personexperience.
 9. The system according to claim 1 wherein the stores can beviewed in real time.
 10. The system according to claim 1 wherein thestores can be viewed in lag time.
 11. The system according to claim 1wherein the feedback is tactile feedback.
 12. The system according toclaim 1 wherein the feedback is electronic feedback.